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EXECUTIVE  SUMMARY 


PROBLEM 

The  SIAP  Implementation  Plan  states  that  the  overarching  objective  of  the  SIAP 
SE  TF  is  "to  identify  incremental  improvements  in  the  ...  SIAP  capability  that  will 
provide  commensurate  incremental  improvements  in  warfighting  capabilities." 
Quantification  of  the  impact  of  improvements  in  SIAP  capability  from  MOPs 
through  attributes  to  MOEs  in  evaluative,  predictive,  and  prescriptive  terms  is  a 
necessary  step  towards  assessment  of  such  improvements. 

OBJECTIVES/PURPOSE 

Complete  the  SIAP  metrics  hierarchy  by  establishing  additional  levels  of 
measurement  in  order  to  provide  the  linkage  between  system  performance  to 
warfighting  benefit.  Specifically: 

-  Define  a  precise  set  of  MOEs  that  measure  warfighting  benefit. 

-  Provide  functional  groupings  of  MOPs  that  measure  system 
performance  as  derived  in  the  TAMD  Integrated  Architecture. 

-  Establish  the  context  for  MOP  measurement. 

-  Discuss  the  relationship  between  MOEs/MOPs  and  SIAP  attributes. 

-  Provide  the  framework  that  enables  analysis  of  the  impacts  of 
incremental  system  level  improvements  on  SIAP  attributes,  and  the 
effect  of  those  improvements  on  MOEs. 

-  Conform  to  definitions  contained  in  the  Theater  Air  and  Missile 
Defense  Capstone  Requirements  Document  (TAMD  CRD)  and 
Combat  Identification  Capstone  Requirements  Document  (CID  CRD) 
definitions. 

APPROACH 

1 )  Establish  joint  consensus  on  a  small  well-defined  set  of  MOEs  that  are  likely 
to  be  affected  by  the  air  picture  quality  as  measured  by  the  SIAP  attributes. 

2)  Provide  a  representative  list  of  MOPs,  partitioned  into  functional  areas  that 
directly  map  from  the  TAMD  Integrated  Architecture,  and  allow  measurement 
of  system  level  performance  based  on  specific  engineering 
changes/improvements.  Use  issues  pertaining  to  specific  data  links  such  as 
Link-16  to  initially  guide  the  MOP  selection  process,  while  providing  sufficient 
flexibility  for  refinement  and  adaptation  of  definitions  to  other  contexts. 

3)  Modify  metrics  as  necessary  as  SIAP  analysis  matures.  Annual  updates  of 
this  technical  report  should  answer  to  further  system-specific  and  data-link- 
specific  issues  that  may  be  raised  in  the  interim. 

FINDINGS 

The  multi-level  description  of  MOPs  and  MOEs  establish  an  understandable, 
disciplined  process  of  tracing  effects  of  system  performance  to  warfighting 
benefits.  It  is  also  acknowledged  that  these  metrics  will  undoubtedly  evolve  over 
time  as  SIAP  analysis  matures. 
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CONCLUSIONS 

The  MOPs  and  MOEs  provide  a  common  framework  for  greater  SIAP  community 
analysis  and  communication  and  directly  support  the  development  of  the  TAMD 
Integrated  Architecture. 


RECOMMENDATIONS 

“Institutionalize”  the  metrics  definitions  and  their  corresponding  hierarchy  within 
the  joint  community.  Update  the  list  of  SIAP  metrics  and  refine  their  definitions 
on  a  periodic  basis  as  warranted  by  the  progress  of  analysis  efforts. 
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1.  INTRODUCTION 


This  report  describes  Measures  of  Effectiveness  (MOEs)  and  Measures  of 
Performance  (MOPs)  to  be  used  in  assessing  the  warfighting  benefits  of  a  Single 
Integrated  Air  Picture  (SIAP)  and  in  controlling  the  system  engineering  process 
by  which  a  SIAP  capability  is  achieved.  Together  with  the  SIAP  attributes,  which 
are  defined  in  SIAP  SE  TF  Technical  Report  2001  -001 ,  these  MOEs  and  MOPs 
form  a  hierarchy  of  metrics  for  evaluating  SIAP  capability,  predicting  the 
performance  of  SIAP-related  improvements  in  terms  of  warfighting  benefit,  and 
prescribing  system  modifications  for  improving  performance.  MOEs  will  be  used 
to  quantify  specific  warfighting  benefits  of  a  SIAP,  and  MOPs  will  be  used  to 
identify  and  characterize  system  performance  for  achieving  a  SIAP.  The  term 
“metric”  used  throughout  this  report  will  refer  generically  to  any  quantitative 
datum  used  in  the  course  of  SIAP  assessments. 

For  this  report,  MOEs,  SIAP  attributes,  and  MOPs  will  be  defined  as 
follows: 

MOE  -  measure  of  operational  success  that  must  be  closely  related 
to  the  objective  of  the  mission  or  operation  being  evaluated.  (DSMC 
Glossary,  2001) 

A  meaningful  MOE  must  be  quantifiable  and  a  measure  to  what  degree  a  mission 
objective  is  achieved. 

SIAP  Attribute  -  measure  of  a  quantifiable  property  of  a  SIAP  that 
is  derived  from  TAMD  and  CID  CRD  requirements  and  associated 
Key  Performance  Parameters  (KPPs). 

MOP  -  measure  of  a  system's  technical  performance,  for  example, 
expressed  as  speed,  payload,  range,  time  on  station,  frequency,  or 
other  distinctly  quantifiable  performance  feature.  (DSMC  Glossary, 

2001) 

As  shown  in  Figure  1 ,  MOEs,  attributes,  and  MOPs  form  a  metrics  hierarchy. 
MOEs  describe,  in  warfighting  terms,  the  benefit  of  achieving  a  SIAP  (note, 
however,  that  there  may  be  motivations  other  than  warfighting  -  for  example, 
commercial  air  applications).  Reduction  in  fratricide  and  increased  weapon 
efficiency  are  examples  of  MOEs  that  are  expected  to  improve  with  improved  air 
picture  quality.  The  SIAP  attributes  provide  a  standard  means  of  characterizing 
the  SIAP.  The  SIAP  attributes  derive  from  the  TAMD  and  CID  CRDs.  Examples 
of  SIAP  attributes  include  completeness,  clarity,  accuracy,  continuity,  and 
commonality.  The  MOPs  quantify  aspects  of  system  and  subsystem 
performance  that  may  be  used  in  the  analysis  of  SIAP  shortfalls  or  the 
prescription  of  SIAP  improvements. 
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Figure  1 .  Relationships  among  SIAP  metrics. 

The  SIAP  SE  TF  uses  a  collection  of  tools  to  define  and  develop  a 
disciplined  joint  system  engineering  approach  and  methodology.  Tools  include 
models,  simulations,  hardware  in  the  loop  (FIWIL)  experiments,  operator  in  the 
loop  (OITL)  experiments,  and  live  exercises.  These  tools  can  be  used  to 
measure  a  wide  variety  of  characteristics  throughout  a  prescribed  metrics 
hierarchy.  Some  tools  provide  a  high  fidelity  assessment  of  system  or 
subsystem  performance  as  a  function  of  engineering  configuration.  These  tools 
expedite  the  determination  of  the  root  cause  of  SIAP  deficiencies.  The  metrics 
that  enter  into  the  quantitative  assessment  process  at  this  level  are  the  SIAP 
MOPs.  The  sensitivities  of  the  SIAP  attributes  to  the  MOPs  associated  with  a 
particular  system  (or  subsystem)  give  an  indication  of  that  system's  contribution 
to  the  SIAP. 

A  suite  of  tools  will  also  be  required  to  characterize  and  assess  the 
warfighting  benefit  derived  from  a  SIAP.  Through  the  use  of  such  tools,  the  SIAP 
attributes  can  be  linked  to  metrics  (MOEs)  that  more  directly  describe  and 
quantify  warfighting  capabilities.  A  few  tools,  such  as  live  exercises,  span  the 
entire  metric  range,  and  can  relate  system/subsystem  level  MOPs  to 
mission/campaign  warfighting  level  MOEs  directly.  Because  there  will  be  fewer 
opportunities  to  employ  live  exercises  due  to  schedule  and  cost  considerations, 
most  tools  will  use  some  degree  of  modeling  and  simulation  and  will  make  the 
linkage  between  MOPs  and  MOEs  through  the  intermediate  level  of  SIAP 
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attributes.  The  ultimate  goal  of  prescribing  engineering  performance 
improvements  which  result  in  significant  warfighting  benefit  translates  analytically 
into  identifying  feasible  changes  in  system-level  MOPs  that  correlate,  through  a 
testing  and  modeling  process,  with  meaningful  improvements  in  MOEs. 

This  paper  defines  MOEs  and  MOPs  and  relates  MOEs,  SIAP  attributes, 
and  MOPs  within  an  overall  metrics  hierarchy.  It  should  be  understood  however, 
that  the  MOEs  and  MOPs  provided  in  this  paper  are  not  exhaustive,  and  will  be 
modified  as  analysis  efforts  dictate.  Changes  will  initially  be  documented  through 
the  development  of  the  system  and  technical  views  (SV  &TV)  of  the  SIAP 
component  of  the  TAMD  Integrated  Architecture,  and  this  report  will  be  updated 
on  an  annual  basis.  The  functional  categories  are  intended  to  remain  fixed,  but 
the  definitions  of  specific  metrics  may  need  to  be  adapted.  This  paper  is  meant 
to  be  limited  in  scope,  and  will  not  make  any  specific  recommendations  as  to  how 
these  metrics  are  to  be  implemented,  or  what  role  they  may  play  in  particular 
models  used  in  the  SIAP  system  engineering  process.  The  summary  overview  of 
that  process  given  in  Section  2.2  clarifies  terminology  and  illustrates  the  roles  of 
the  different  levels  of  metrics.  Details  of  metrics  implementation  and  a 
quantitative  treatment  of  some  of  the  modeling  issues  will  be  the  subjects  of 
SIAP  SE  TF  Technical  Reports  2001-003  and  2001  -004,  with  which  this 
document  will  eventually  be  linked. 


2.  THE  SIAP  METRICS  HIERARCHY 

The  "SIAP  Metrics  Hierarchy"  refers  to  the  SIAP  metrics  structure 
described  above,  together  with  a  further  sub-classification  within  the  MOE  and 
MOP  levels.  The  sub-classification  is  a  means  to  identify  the  level  at  which 
warfighting  effectiveness  is  considered  (in  the  case  of  MOEs),  or  the  particular 
system-level  function  being  assessed  (in  the  case  of  MOPs).  The  definitions 
provided  in  the  next  section  are  meant  to  provide  a  standard  for  SIAP  system 
engineering  usage. 

2.1  Definition  of  Terms  and  Assumptions 

The  following  key  terms  are  either  used  in  this  report  or  are  generally 
relevant  to  the  discussion  of  the  SIAP  metrics  hierarchy: 

Functional  Area  -  a  subcategory  of  MOPs  measuring  related 
aspects  of  SIAP  system  performance  as  defined  by  the  Systems 
View  (SV)  of  the  SIAP  component  of  the  TAMD  Integrated 
Architecture. 

Strategic  Level  of  War  -  The  level  of  war  at  which  a  nation,  often  as 
a  member  of  a  group  of  nations,  determines  national  or 
multinational  (alliance  or  coalition)  security  objectives  and 
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guidance,  and  develops  and  uses  national  resources  to  accomplish 
those  objectives.  Activities  at  this  level  establish  national  and 
multinational  military  objectives;  sequence  initiatives;  define  limits 
and  assess  risks  for  the  use  of  military  and  other  instruments  of 
power;  develop  global  or  theater  war  plans  to  achieve  those 
objectives;  and  provide  military  forces  and  other  capabilities  in 
accordance  with  strategic  plans.  (JP  1  -02,  2001 ) 

Operational  Level  of  War  -  The  level  of  war  at  which  campaigns 
and  major  operations  are  planned,  conducted,  and  sustained  to 
accomplish  strategic  objectives  within  theaters  or  areas  of 
operations.  Activities  at  this  level  link  tactics  and  strategy  by 
establishing  operational  objectives  needed  to  accomplish  the 
strategic  objectives,  sequencing  events  to  achieve  the  operational 
objectives,  initiating  actions,  and  applying  resources  to  bring  about 
and  sustain  these  events.  These  activities  imply  a  broader 
dimension  of  time  or  space  than  do  tactics;  they  ensure  the  logistic 
and  administrative  support  of  tactical  forces,  and  provide  the  means 
by  which  tactical  successes  are  exploited  to  achieve  strategic 
objectives.  (JP  1  -02,  2001) 

Tactical  Level  of  War  -  The  level  of  war  at  which  battles  and 
engagements  are  planned  and  executed  to  accomplish  military 
objectives  assigned  to  tactical  units  or  task  forces.  Activities  at  this 
level  focus  on  the  ordered  arrangement  and  maneuver  of  combat 
elements  in  relation  to  each  other  and  to  the  enemy  to  achieve 
combat  objectives.  (JP  1  -02,  2001) 

Campaign  -  A  series  of  related  military  operations  aimed  at 
accomplishing  a  strategic  or  operational  objective  within  a  given 
time  and  space.  (JP  1  -02,  2001 ) 

Operation  -  1 .  A  military  action  or  the  carrying  out  of  a  strategic, 
operational,  tactical,  service,  training,  or  administrative  military 
mission.  2.  The  process  of  carrying  on  combat,  including, 
movement,  supply,  attack,  defense,  and  maneuvers  needed  to  gain 
the  objectives  of  any  battle  or  campaign.  (JP  1  -02,  2001 ) 

Engagement  -  In  air  defense,  an  attack  with  guns  or  air-to-air 
missiles  by  an  interceptor  aircraft,  or  the  launch  of  an  air  defense 
missile  by  air  defense  artillery  and  the  missile's  subsequent  travel 
to  intercept.  (JP  1  -02,  2001 ) 

The  position  of  the  various  categories  of  SIAP  metrics  within  the  metrics 
hierarchy  is  suggested  by  Figure  2. 
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Figure  2.  Metrics  hierarchy. 

2.2  Metrics  Traceability  Process 

The  "metrics  traceability  process"  refers  to  the  process  of  establishing 
quantitative  functional  relationships  between  metrics  at  different  levels  in  the 
hierarchy,  but  most  especially  between  MOPs  directly  associated  with  testable 
system  performance  characteristics  and  MOEs  quantifying  warfighting  capability. 
With  such  a  process  in  place,  the  impact  of  incremental  improvements  at  the 
system  MOP  level  should  be  observed  (if  such  functional  relationships  exist)  at 
the  higher  levels.  The  goal  is  to  identify  system  the  performance  impacts  on 
warfighting  capability. 

A  complete  characterization  of  all  such  relationships  is  often  not  practical 
(the  number  of  metrics  being  enormous  and  their  interrelationships  extremely 
complex).  Therefore,  a  practical  traceability  study  will  typically  involve  just  a  few 
representative  metrics  from  two  or  more  subcategories  within  the  hierarchy. 
MOEs  will  be  chosen  to  capture  the  most  relevant  indications  of  mission  or 
campaign  outcome  for  the  operational  scenario  under  consideration.  Given  the 
likelihood  of  insufficient  data  to  make  a  direct  connection  with  any  system-level 
MOPs,  the  study  might  begin  with  a  preliminary  "top-down"  scoping  analysis. 

The  SIAP  attributes  believed  to  have  the  strongest  effect  on  the  MOEs  of  interest 
would  be  identified,  along  with  the  principal  MOPs  on  which  these  attributes 
appear  to  depend.  The  result  would  be  a  preliminary  (but  quantitative)  functional 
dependence  of  selected  SIAP  attributes  on  a  few  key  MOPs.  It  should  be  noted 
that  such  cause-effect  relationships  between  MOPs  and  SIAP  attribute  measures 
do  not  generally  follow  from  the  definitions  of  the  metrics,  but  will  need  to  be 
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supported  (even  in  their  preliminary  form)  through  some  degree  of  testing  and 
modeling.  This  process  has  been  characterized  as  “Root-Cause”  analysis. 

A  separate  modeling  procedure  will  be  used  to  quantify  the  dependence  of 
the  key  MOEs  on  the  attributes  in  question.  Combining  the  two  analyses  yields  a 
(probably  implicit)  quantitative  functional  connection  between  a  few  MOEs  and  a 
few  MOPs.  Finally,  discernment  of  any  gaps  in  the  analysis  process  (where  the 
quantitative  links  are  believed  to  be  tenuous)  could  suggest  further  tests  to  obtain 
the  data  needed  to  strengthen  the  claims  of  the  analysis.  Several  studies  of  this 
sort,  focusing  on  different  groups  of  MOPs,  would  be  needed  to  arrive  at  a  firm 
basis  for  comparison  of  system  engineering  alternatives  with  regard  to 
warfighting  payoffs. 

A  study  incorporating  an  example  of  the  metrics  traceability  process  is 
being  undertaken  concurrently  with  the  writing  of  this  report.  It  focuses  on  the 
functional  area  of  data  registration  in  the  context  of  a  mission-level  vignette.  The 
results  of  this  study  are  to  be  published  in  SIAP  SE  TF  Technical  Report  2001  - 
004. 


3.  MOEs 

3.1  Definition  of  Terms  and  Assumptions 

Specific  definitions  relating  to  the  MOEs  are  as  follows: 

Commit  -  The  process  of  committing  one  or  more  air  interceptors 
or  surface-to-air  missiles  for  interception  against  a  target  track.  (JP 
1-02,  2001) 

Decision  Time  -  Time  from  establishment  of  firm  track  (i.e.,  the  time 
at  which  the  track  first  acquires  a  track  number  in  a  participant's 
central  track  store)  to  the  commitment.  While  this  measure  does 
not  specifically  enter  into  the  discussion  of  MOEs  presented  in  the 
following  section,  decision  time  is  a  frequently  used  concept,  and  it 
is  important  to  distinguish  it  from  the  other  time  measures 
introduced  in  this  report. 

3.2  Warfighting  Capabiiity 

As  explained  in  the  Introduction,  the  scope  of  this  report  does  not  allow  for 
an  exhaustive  treatment  of  metrics,  nor  is  such  a  treatment  warranted.  The 
selection  of  MOEs,  in  particular,  should  take  into  account  details  of  the  scenario 
being  analyzed.  These  details  are  still  under  development  (scenarios  will  be 
described  in  the  SIAP  Common  Reference  Scenario  Technical  Report,  2001-005, 
in  preparation).  However,  the  Joint  Theater  Air  and  Missile  Defense 
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Organization  (JTAMDO)  has  endorsed  a  representative  list  of  MOEs,  which  can 
be  expected  to  cover  most  warfighting  capabilities  that  are  relevant  to  air  warfare, 
irrespective  of  the  scenario  details.  It  is  the  intent  of  this  section  to  present  these 
generally  applicable  MOEs,  and  to  provide  some  guidance  on  their  intended  use 
in  traceability  analysis.  To  this  end,  examples  of  one  or  more  specific 
quantitative  metrics  are  provided  for  each  MOE,  with  the  understanding  that  the 
definitions  of  these  specific  metrics  may  change  slightly  in  accordance  with  the 
scenario  details  or  the  nature  of  the  analysis  undertaken.  It  should  be  noted  that 
MOEs  are  not  exclusively  SIAP  measures,  and  most  of  the  ones  described  in  this 
section  have  been  used  to  assess  warfighting  capability  in  other  contexts. 

As  discussed  in  Section  2.1 ,  the  U.S.  Department  of  Defense  (DoD) 
provides  definitions  of  engagement,  operation,  and  campaign  that  can  be  used 
for  categorizing  MOEs.  Table  1  provides  MOEs,  endorsed  by  JTAMDO,  many  of 
which  are  applicable  at  multiple  levels,  and  definitions  of  one  or  more  specific 
quantitative  metrics  pertaining  to  each  MOE. 


MOE 

Metric  Examples 

Leakers 

Total  number  of  Hostile  weapon  systems  (manned  or 
unmanned)  that  reach  their  ordnance  release  points:  by  type 

Hostile  Attrition 

Total  number  of  Hostile  targets  killed:  by  target  type 

Friendly 

Attrition 

Total  number  of  Friendly  targets  killed:  by  asset  type 

Fratricide 

Total  number  of  Friendly  targets  killed  by  Friendly  forces:  by 
asset  type  and  shooter  type 

Total  number  of  Neutral  targets  killed  by  Friendly  forces:  by 
asset  type  and  shooter  type 

Weapon 

Expenditures 

Total  number  of  weapons  expended:  by  type 

C2 

Total  number  of  engagements  ordered:  by  type  and  by  target 

Blue  sortie  rates  ordered:  by  mission,  force,  and  function 

Battlespace 

Location/Time  of  weapon  commit 

Time/Distance  from  initial  detection  to  commit 

Time/Distance  from  ID  (other  than  unknown)  to  commit 

Location/Time  of  intercept  (engagement) 

Table  1.  MOEs. 


While  the  names  given  to  the  MOEs  themselves  are  widely  used  terms  in 
the  defense  community,  there  has  been  some  inconsistency  in  the  use  of  these 
terms  specifically  as  MOEs.  Therefore,  in  addition  to  the  metric  examples  given 
in  Table  1  to  illustrate  the  intended  use,  the  following  qualitative  discussion  of  the 
terminology  may  also  help  to  clarify  distinctions  between  the  different  MOEs. 

Leakers  and  battlespace  MOEs  both  capture  the  extent  to  which  threat  objects 
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penetrate  to  specific  points,  or  into  specific  spatial  regions,  but  they  do  so 
through  different  measurement  approaches.  The  number  of  leakers  of  a  certain 
type  simply  counts  the  number  of  hostile  aerospace  objects  of  that  type  which 
penetrate  far  enough  to  accomplish  the  enemy's  objective  -  for  example,  the 
number  of  attacking  enemy  planes  which  reach  their  designated  missile  (or 
bomb)  release  points.  The  battlespace  metrics,  on  the  other  hand,  quantify  in 
spatial  and  temporal  terms  exactly  how  far  the  threat  objects  penetrate,  without 
regard  to  their  effectiveness.  The  battlespace  may  either  be  quantified  in 
absolute  terms,  or  relative  to  where  (when)  the  threat  is  detected.  The  example 
metrics  provided  are  typical,  but  not  the  only  possibilities.  Attrition  (hostile  or 
friendly)  is  a  quantification  of  damage  done  in  terms  of  targets  killed,  "killing"  of  a 
target/asset  for  most  purposes  being  understood  as  rendering  the  target/asset 
inoperable  for  the  duration  of  the  campaign  (although  the  scenario  details  might 
allow  for  temporary  kills,  in  which  case  campaign-level  attrition  might  only  count 
the  permanently  killed  targets).  Fratricide  is  a  measure  of  friendly  (or  neutral,  if 
applicable)  attrition  attributable  to  friendly  fire.  Weapons  expenditure  is  fairly 
self-explanatory,  being  an  absolute  count  of  weapons  used  by  type.  A  derived 
measure  of  (friendly)  weapons  efficiency  is  obtained  by  comparing  weapons 
expenditure  to  hostile  attrition  (for  example,  targets  killed  per  weapon  used,  over 
a  particular  engagement).  The  command  and  control  (C2)  metrics  quantify  the 
effectiveness  of  friendly  force  decision-making,  or  the  number  of  command 
results  of  a  type  generally  considered  indicative  of  friendly  force  effectiveness 
(engagements,  sorties).  These  metrics  attempt  to  capture  relevant  higher-level 
consequences  of  the  other  MOEs  (ability  to  fly  sorties,  for  example,  indicates 
attainment  of  a  certain  degree  of  air  superiority  -  an  expected  consequence  of  air 
defense  effectiveness  as  indicated  by  other  MOEs). 

Decision  time,  defined  in  Section  3.1 ,  may  be  regarded  as  a  composite 
measure  derivable  from  the  battlespace  MOE  metrics  and  system-level  data 
(track  initiation  time). 


4.  SIAPMOPs 

4.1  Definition  of  Terms  and  Assumptions 

The  following  special  terms  are  used  when  defining  MOPs.  Some  of  this 
terminology  overlaps  with  that  of  SIAP  SE  TF  Technical  Report  2001  -001 ,  and  is 
included  here  for  completeness.  Issues  pertaining  to  specific  data  links  such  as 
Link-1 6  (and,  to  a  lesser  extent,  Link-1 1 )  have  been  used  to  give  the  MOP 
definition  process  some  initial  guidance,  hence  the  Link-1 6  focus  in  much  of  this 
terminology. 

Common  Reference  Scenario  (CRS)  -  a  Defense  Planning 
Guidance  (DPG)-based  operational  context  for  SIAP  assessments, 
providing  a  consistent  baseline  for  evaluating  current  performance 
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and  proposed  improvements.  The  CRS  is  consistent  with  the 
concepts  of  a  scenario-based  design,  that  is  tailored  to  the  SIAP 
component  of  the  TAMD  Integrated  Architecture. 

Area  of  Interest  (API)  -  "that  area  of  concern  to  the  commander, 
including  the  area  of  influence,  areas  adjacent  thereto,  and 
extending  into  enemy  territory  to  the  objectives  of  current  or 
planned  operations.  The  area  also  includes  areas  occupied  by 
enemy  forces  who  could  jeopardize  accomplishment  of  the  mission" 
(JP  1  -02,  2001 ).  The  definition  of  an  AOI  will  be  further  refined  in 
the  CRS  to  bound  the  area  within  which  SIAP  attributes  will  be 
evaluated.  The  AOIs  in  the  CRS  may  be  broken  down  into  limited 
participant/mission  specific  AOIs  for  the  assessment  of  attributes 
for  a  given  participant  or  mission. 

Reporting  Criteria  -  will  be  prescribed  in  the  SV  of  the  SIAP 
component  of  the  TAMD  Integrated  Architecture. 

Object  -  any  airborne  missile,  aircraft,  large  caliber  rocket,  or  other 
tactically  significant  item  in  the  AOI  meeting  reporting  criteria. 

Track  -  "(1 )  the  graphic  and/or  alphanumeric  representation  of  an 
object,  point,  or  bearing  whose  position  and/or  characteristics  are 
collated  from  sensors  and/or  other  data  sources;  (2)  a  collated  set 
of  data... associated  with  a  track  number  for  the  purpose  of 
representing  the  position  and/or  characteristics  of  a  specific  object, 
point,  or  bearing"  (MIL-STD-6016A,  1999).  For  SIAP  assessment 
purposes,  a  track  is  understood  as  an  actionable  track,  not  to 
include  tentative  tracks  or  clutter  tracks. 

Assigned  Track  -  an  after-the-fact  analysis  of  track  data  from  an 
exercise  or  simulation,  a  track  which  meets  a  consistent  condition 
for  association  with  an  object  (addressed  in  SIAP  SE  TF  Technical 
Report  2001-003).  The  assignment  of  tracks  to  objects  may  be 
time  dependent. 

Lost  Track  -  a  track  which  is  not  being  updated  on  the  basis  of 
current  sensor  data,  but  which  has  not  been  deleted  by  the  system. 

Reporting  Responsibility  (R2)  -  "the  requirement  for  the  interface 
unit  with  the  best  positional  data  on  a  track  to  transmit  track  data  on 
the  interface"  (MIL-STD-6016A,  1999).  Reporting  responsibility 
criteria  may  later  be  expanded  to  consideration  of  the  best  full 
kinematic  data. 
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Track  Quality  (TQ)  -  "A  measure  of  the  reliability  of  the  positional 
information  of  a  reported  track"  (MIL -STD -60 16 A,  1999). 

Gridlock  -  "a  procedure  for  determining  data  registration  corrective 
values  (pads)  by  computing  remote  tracks  received  from  a 
designated  reference  unit  to  local  data"  (MIL-STD-601 1  A,  1997). 

Correctness  -  (when  used  as  part  of  the  name  of  an  MOP)  a  measure  of 
system  output  given  system  data  and  the  algorithm  that  generates  the 
system  output.  Correctness-based  MOPs  in  this  report  compare  a 
system's  error  or  quality  estimates  with  estimates  produced  offline  using 
the  same  input  data  and  the  algorithm  that  the  system  is  supposed  to 
implement  (does  not  address  the  quality  of  the  data  or  algorithm  used,  and 
does  not  depend  on  truth  data). 

Consistency  -  (when  used  as  part  of  the  name  of  an  MOP)  a  measure  of 
how  well  a  quality  or  error  estimate  represents  the  actual  errors  of  the 
system  (requires  truth  data  to  evaluate  actual  errors). 

More  general  defense  terms  and  acronyms  used  within  the  definitions  in 
this  section  can  be  found  in  MIL-STD-601 6A  (1 999)  or  Joint  Publication  1  -02 
(2001). 

4.2  SIAP  Functional  Areas  and  Sample  MOPs 

It  is  necessary  to  examine  MOPs  as  they  relate  to  the  functional  areas 
defined  in  the  SV  of  the  SIAP  component  of  the  TAMD  Integrated  Architecture. 
When  considering,  for  example,  a  functional  area  regarding  combat  identification, 
a  number  of  distinct  MOPs,  including  ID  assessment  (a  comparison  of  ID 
declarations  with  truth),  and  ID  program  performance  (an  evaluation  of  whether 
an  ID  procedure  is  functioning  as  designed)  may  be  relevant.  This  section  gives 
examples  of  functional  areas  and  qualitative  definitions  of  some  of  the  associated 
MOPs  as  well  as  additional  MOPs  that  stand-alone.  These  MOPs  are  also  listed 
in  tabular  form  for  convenient  reference  in  Appendix  A.  A  list  of  critical  variables 
needed  to  compute  these,  and  possibly  other,  MOPs  is  supplied  in  Appendix  B. 

While  it  is  intended  that  this  list  of  MOPs  should  remain  stable  with  regard 
to  the  functional  areas  and  the  qualitative  aspects  of  the  individual  definitions, 
some  of  the  definitions  will  be  refined  mathematically  as  assessment  needs  for 
specific  systems  are  clarified  in  specific  contexts  (live  tests,  HWIL,  etc.).  The 
needs  of  the  data  analyst  in  context  should  dictate  the  final  definition  or  formula 
used.  However,  to  fulfill  the  SIAP  SE  TP's  objective  of  establishing  a 
standardized  language  and  consistent  use  of  metrics  across  contexts,  it  is 
expected  that  all  such  refinements,  as  well  as  proposals  for  additional  SIAP 
MOPs,  will  be  addressed  by  the  SIAP  Analysis  Team  (SAT)  during  Data 
Management  Analysis  Plan  (DMAP)  preparation  of  specific  analysis  events. 

Page  1 8 
Version  1 .0 
October  25,  2001 


1.  TIME 

Time  synchronization  -  The  time  history,  referenced  to  Coordinated  Universal 
Time  (UTC),  of  the  instantaneous  difference  between  a  participant's  clock  time 
and  UTC. 

Element  data  exchange  latency  -  The  time  required  for  a  message/data  event  in 
an  element  to  trigger  the  desired  response.  There  are  numerous  latencies  that 
could  be  considered,  including  time  from  measurements  to  local  track  update, 
time  from  local  track  update  to  transfer  to  radio  for  transmission,  time  from 
transfer  to  radio  for  transmission  to  transmission,  time  from  receipt  of  message  to 
transfer  to  local  track  store,  etc. 

Time  to  get  track  report  on  the  air  -  The  time  from  establishment  of  firm  track 
(i.e.,  first  appearance  of  reported  track  in  some  participant's  CTS)  to  initiation  of 
the  R2  process. 

Lost  track  persistence  -  The  duration  of  time  a  lost  track  remains  displayable 
before  either  being  designated  as  dropped  and  being  deleted  from  the  network, 
or  being  purged  from  the  local  track  stores. 

2.  SENSORS/TRACKERS 

Sensor  detection  range  -  The  slant  range  at  which  the  energy  received  from  an 
observed  object  or  unresolved  cluster  of  objects  exceeds  a  detection  threshold. 

Sensor  error  -  The  instantaneous  error  during  the  entire  run  between  an  object’s 
true  kinematic  data  and  the  locally  held  sensor  kinematic  data  minus  any 
navigation  errors.  Sensor  error  may  be  broken  down  into: 

-  Measurement  noise  errors  (which  by  definition  have  zero  mean).  These  may 
be  characterized  by  (correctness  and  consistency  of)  measurement  error 
covariance  over  the  applicable  subset  of  azimuth,  elevation,  range  and  range- 
rate  measurement  components  for  a  nominal  signal-to-noise  ratio. 

-  Residual  biases. 

A  further  breakdown  by  source  of  error  is  also  admissible. 

TQ  correctness  -  The  correctness  with  which  a  unit  calculates  track  quality  (TQ) 
based  upon  MIL-STD-6016A. 

TQ  consistency-  The  consistency  of  reported  TQ  with  sensor  error  data. 

3.  DATA  CONNECTIVITY 

Time  distribution  of  connectivity  failures  (radio-to-radio)  -  Measure  of  the 
distribution  of  time  between  failures  in  connectivity  (radio-to-radio)  between 
participating  units. 
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Time  distribution  of  duration  of  connectivity  failures  (radio-to-radio)  -  Measure  of 
the  distribution  of  duration  of  the  connectivity  failures  (radio-to-radio)  between 
participating  units. 

Time  distribution  of  connectivity  failures  (track  store-to-track  store)  -  Measure  of 
the  distribution  of  time  between  failures  in  connectivity  (track  store-to-track  store) 
between  participating  units. 

Time  distribution  of  duration  of  connectivity  failures  (track  store-to-track  store)  - 
Measure  of  the  distribution  of  duration  of  the  connectivity  failures  (track  store-to- 
track  store)  between  participating  units. 

TADIL  update  rates  -  The  rate  at  which  periodic  TADIL  message 
occurrences  are  generated. 

4.  DATA  REGISTRATION 

Geodetic  registration 

Navigation  error(s)  -  The  instantaneous  error  during  the  entire  run  between  a 
unit’s  true  position,  velocity,  and  body  attitude  (based  on  WGS-84)  and  the 
navigation  estimates.  As  with  sensor  error,  may  be  broken  down  by  type  or 
source,  and  separate  measures  may  be  defined  for  correctness  and  consistency. 

Navigation  Qpn  correctness  -  The  correctness  with  which  a  unit  calculates  its 
Geodetic  Position  Quality  (Qpg),  based  upon  the  defining  algorithm. 

Navigation  Qpn  consistency  -  The  consistency  of  reported  Geodetic  Position 
Quality  (Qpg)  with  navigation  error  data. 

lU  Registration 

lU  registration  error  -  Residual  errors  in  the  estimates  from  lU  registration  of 
relative  horizontal  position  between  pairs  of  sensor  platforms  and  the  pair-wise 
relative  azimuth  alignment  between  sensor  apertures  on  those  platforms  (i.e.  a 
measure  of  compliance  with  MIL-STD-601 6A  lU  registration). 

lU  registration  error  covariance  consistency  -  A  measure,  such  as  a  normalized 
chi-squared  statistic,  of  the  extent  to  which  lU  registration  residual  error 
covariance  is  a  true  representation  of  the  pair-wise  relative  residual  biases. 

Sensor  Registration 

Sensor  registration  error  -  Residual  errors  in  discrete  or  roll-up  corrections 
estimated  by  sensor  registration.  This  MQP  may  be  represented,  if  appropriate, 
by  separate  measures  for  individual  sources  of  error. 

Sensor  registration  error  covariance  consistency  -  A  measure,  such  as  a 
normalized  chi-squared  statistic,  of  the  extent  to  which  sensor  registration 
residual  error  covariance  is  a  true  representation  of  the  residual  bias  error. 
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Sensor  Gridlock 

Network-wide  absolute  sensor  gridlock  error  -  Residual  errors  in  corrections 
estimated  by  network-wide  absolute  sensor  gridlocking  of  the  geodetic  horizontal 
position  and  altitude  of  each  sensor  platform  in  the  network,  and  of  the  geodetic 
angular  alignment  of  each  participating  sensor  aperture  on  those  platforms. 

Network-wide  absolute  sensor  gridlock  error  covariance  consistency  -  A 
measure,  such  as  a  normalized  chi-squared  statistic,  of  the  extent  to  which 
network-wide  absolute  sensor  gridlock  residual  error  covariance  is  a  true 
representation  of  the  absolute  residual  biases. 

Data  Processing 

Computational  errors  -  A  class  of  measures  intended  primarily  to  capture  the 
effects  of  inaccurate  algorithms  or  poor  algorithm  implementation.  A  comparison 
of  the  result  of  any  system-generated  computation  (such  as  a  coordinate 
transformation)  with  the  result  of  the  same  computation  performed  accurately 
offline  may  serve  as  the  MOP. 

5.  CORRELATION/DECORRELATION 

Correct  correlation  rate  -  Measure  of  the  rate  of  correct  correlations  performed 
by  the  correlation  algorithm. 

Correct  non-correlation  rate  -  Measure  of  the  rate  of  correct  non-correlations 
performed  by  the  correlation  algorithm. 

Incorrect  non-correlation  rate  -  Measure  of  the  rate  of  incorrect  non-correlations 
performed  by  the  correlation  algorithm. 

False  correlation  rate  -  Measure  of  the  rate  of  false  correlations  performed  by  the 
correlation  algorithm. 

6.  REPORTING  RESPONSIBILITY 

R2  correctness  -  Measure  of  whether  R2  was  attributed  correctly  according  to 
the  rules  of  assuming  reporting  responsibility  as  defined  in  MIL-STD-601 6A. 

7.  COMBAT  ID 

ID  program  performance  -  Comparison  of  the  ID  declaration  on  a  track  with  the 
ID  that  would  be  applied  based  on  the  combat  identification  information  available 
and  the  rules  of  engagement.  For  this,  and  the  other  combat  ID  "performance" 
MOPs  which  follow,  the  means  of  assessing  the  information  available  will  depend 
upon  the  testing/evaluation  context. 

ID  assessment  -  The  measure  of  whether  the  ID  is  incorrect,  unknown,  or 
correct,  for  an  assigned  track,  as  determined  through  ID  assessment  rules  which 
are  linked  to  the  scenario  under  investigation  (cf.  discussion  of  the  ID  attributes  in 
SIAP  SE  TF  Technical  Report  2001  -001 ). 
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Category  program  performance  -  An  evaluation  of  how  well  each  unit’s  program 
performs  environmental  category  (surface,  land,  air,  space)  classifications,  as 
determined  by  all  relevant  specifications  and  documentation. 

Category  assessment  -  The  measure  of  correctness  of  environmental  category 
classification. 

Target  class/type  program  performance  -  An  evaluation  of  how  well  each  unit’s 
program  performs  target  class  and  type  identifications,  as  determined  by  all 
relevant  specifications  and  documentation. 

Target  class/type  assessment  -  The  measure  of  correctness  of  assessment  of 
target  class  and  type. 

8.  CAPACITY 

Design  capacity  usage  -  Percentage  of  design  capacity  used,  and  to  the  extent 
possible,  broken  down  by  subnet  and/or  network  participation  group  (NPG)  by 
participants  over  time. 

Number  of  tracks  deleted  due  to  storage  limits  -  Total  number  of  tracks  deleted 
from  track  files  due  to  system  storage  limitations. 

Element  update  rates  -  The  rate  at  which  periodic  element  message  occurrences 
are  generated. 

9.  FORMATION  TRACKING 

Fraction  of  tracks  which  are  formation  tracks  -  The  instantaneous  ratio  of  the 
number  of  formation  tracks  (with  strength  >1  or  unspecified)  to  the  total  number 
of  tracks  (measured  at  each  participant). 

Formation  strength  to  tracks  ratio  -  The  instantaneous  ratio  of  the  sum  of 
strengths  of  all  tracks  to  the  number  of  tracks  (measured  at  each  participant). 

10.  OTHERS 

Data  update  reguest  generation  -  The  accuracy  with  which  Data  Update 
Requests  (DURs)  are  generated  and  transmitted. 

Data  update  reguest  response  -  The  accuracy  with  which  DURs  are  received 
and  processed,  and  the  response  message  formulated,  generated,  and 
transmitted. 

Drop  track  generation  and  processing  -  The  accuracy  with  which  Drop  Track 
Reports  (DTRs)  are  generated,  transmitted,  received  and  processed. 
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Unit  status  generation  and  processing  -  The  accuracy  with  which  unit  status 
messages  are  generated,  transmitted,  received  and  processed. 


Engagement  status  generation  and  processing  -  The  accuracy  with  which 
Engagement  Status  is  generated,  transmitted,  received,  and  processed. 

Controlling  unit  report  generation  and  processing  -  The  accuracy  with  which 
Controlling  Unit  Reports  (CURs)  are  generated,  transmitted,  received,  and 
processed. 

Force  order  generation  and  processing  -  The  accuracy  with  which  Force  Orders 
(FOs)  are  generated,  transmitted,  received,  and  processed. 

Change  data  order  generation  and  processing  -  The  accuracy  with  which 
Change  Data  Orders  (CDOs)  are  generated,  transmitted,  received,  and 
processed. 

Track  commonality  -  A  Boolean  measure,  describing  whether  a  track  is  held  by 
all  participants,  satisfying  some  given  constraints  pertaining  to  consistency  of 
positional  and  combat  ID  data.  May  be  restricted  to  a  particular  group  of 
participants  (for  example,  a  particular  NPG,  or  a  group  of  participants  involved  in 
a  launch-on-remote  engagement  of  the  specified  track). 


As  noted  in  Section  2.2,  the  MOPs  provided  do  not  necessarily  relate 
directly  to  the  SIAP  attributes  by  virtue  of  their  definitions.  A  limited  number  of 
MOPs,  however,  are  directly  involved  in  the  computation  of  SIAP  attributes.  The 
remaining  MOPs  are  expected  to  indirectly  relate  to  SIAP  attributes  and 
contribute  to  the  SIAP  system  engineering  process  through  modeling  of 
proposed  system  improvements,  or  through  the  experimental  investigation  of  root 
cause  issues.  The  class  of  MOPs  on  which  the  SIAP  attributes  are  explicitly 
dependent  can  be  regarded  as  a  subset  of  the  critical  variables  needed  to 
evaluate  the  attributes.  Since  the  details  of  the  functional  relationships  involved 
depend  on  the  assignment  procedure  used,  this  subject  will  be  taken  up  more 
thoroughly  under  the  discussion  of  assignment  in  SIAP  SE  TF  Technical  Report 
2001  -003,  Appendix  A. 


5.  SUMMARY 

Initial  lists  of  MOEs  and  MOPs  have  been  provided,  which  along  with  the 
SIAP  attributes  previously  defined,  will  be  utilized  to  predict,  evaluate,  and 
prescribe  engineering  changes  that  result  in  improved  warfighting  capability.  The 
metrics  exist  at  multiple  levels  in  the  overall  process  of  tracing  system-level 
improvements  to  warfighting  benefits.  MOPs  are  considered  measures  at  the 
engineering  level,  a  level  below  SIAP  attributes;  that  is,  they  will  either  directly  or 
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indirectly  impact  SIAP  attributes.  MOEs  are  at  the  warfighting  level,  a  level 
above  SIAP  attributes.  MOEs  are  expected  to  functionally  depend  on  one  or 
more  SIAP  attribute.  The  consolidated  metrics  are  also  in  direct  support  of  the 
TAMD  Integrated  Architecture,  and  provide  the  framework  for  analysis  across  all 
venues,  including:  M&S,  HWIL,  OITL,  as  well  as  live  exercises.  It  is  understood 
that  modifications  will  be  necessary  as  dictated  by  future  analysis  and  provisions 
for  refinement  have  been  noted. 
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APPENDIX  A 


Table  of  SIAP  Functional  Areas  and  MOPs 


Functional  Area 


TIME 


SENSORS/TRACKERS 


DATA  CONNECTIVITY 


DATA  REGISTRATION 


Geodetic  Reqistration 


MOP 


Time  synchronization 


Element  data  exchange  latenc 


Time  to  get  track  report  on  the  air 


Lost  track  persistence 


Sensor  detection  range 


Sensor  error 


TQ  correctness 


TQ  consistenc 


Time  distribution  of  connectivity  failures  (radio-to- 
radio) 


Time  distribution  of  duration  of  connectivity  failures 
(radio-to-radio) 


Time  distribution  of  connectivity  failures  (track  store- 
to-track  store) 


Time  distribution  of  duration  of  connectivity  failures 
(track  store-to-track  store) 


TADIL  update  rates 


Navigation  error(s) 


Navigation  Qnn  correctness 


Navigation  Qnn  consistenc 


lU  registration  error 


lU  registration  error  covariance  consistenc 


Sensor  registration  error 


Sensor  registration  error  covariance  consistenc 


Network-wide  absolute  sensor  gridlock  error 


Network-wide  absolute  sensor  gridlock  error 
covariance  consistenc 


Computational  Errors 


Table  A-1  Functional  areas  and  sample  MOPs.  (continued  next  page) 
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Table  A-1  Functional  areas  and  sample  MOPs.  (Continued) 
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APPENDIX  B 


MOP  Critical  Variables 

This  appendix  lists  variables  critical  to  computing  the  MOPs  listed  in 
Appendix  A.  They  are  numbered  for  convenience.  For  MOPs  depending  upon 
an  assignment  of  tracks  to  objects,  an  appropriate  tracks-to -truth  assignment 
procedure  must  also  be  carried  out  (as  described  in  SIAP  SE  TF  Technical 
Report  2001-003).  For  such  MOPs  it  is  assumed  that,  in  addition  to  the 
quantities  specifically  listed  below,  all  track  data  required  for  the  assignment 
procedure  is  available. 

1 .  System/ID  declaration  (friend,  hostile,  unknown,  etc)  on  each  track  at  each 
evaluation  time. 

2.  System  class  and  type  declaration  on  each  track  at  each  evaluation  time. 

3.  Difference  between  own  unit’s  navigation  measure  and  WGS-84. 

4.  Difference  between  inertial  navigation  measure  and  WGS-84. 

5.  Difference  between  JTIDS  navigational  measure  and  WGS-84. 

6.  Difference  between  own  unit's  clock  time  and  UTC/USNO. 

7.  Total  number  of  correlation  messages. 

8.  Total  number  of  drop  track  messages. 

9.  Total  number  of  PPLI  reports. 

1 0.  Total  number  of  track  management  messages. 

1 1 .  Total  number  of  formation  tracks. 

1 2.  Number  of  timeslot  reallocations  executed.  (This  and  the  five  preceding 
are  often  regarded,  among  others,  as  alternative  measures  of  "network 
loading."  In  the  approach  of  this  report,  these  quantities  are  indirectly 
reflected  in  MOPs  across  several  functional  areas: 

Correlation/Decorrelation,  Capacity,  Formation  Tracking,  and  Others.) 

1 3.  Proportion  of  real  objects  meeting  reporting  criteria  that  are  held  as  a 
declared  track  at  each  scheduled  scoring  time  in  the  scenario. 

1 4.  Time  at  which  a  particular  object  meeting  reporting  criteria  has  a  valid 
declared  track  (track  initiation  time). 
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1 5.  For  real  objects  meeting  reporting  criteria,  the  cumulative  number  of 
switches  (not  counting  breaks)  of  tracks  for  particular  objects  and 
averaged  across  all  objects  by  time  t  into  the  scenario. 

1 6.  For  real  objects  meeting  reporting  criteria,  the  cumulative  number  of 
breaks  of  tracks  for  particular  objects  and  averaged  across  all  objects  by 
time  t  into  the  scenario. 

1 7.  Sensor  aperture  effective  refractivity  at  mean  sea  level  (for  refractive 
corrections). 

1 8.  Sensor  aperture  range  measurement  offset  bias. 

1 9.  Sensor  aperture  range  measurement  scale  factor  bias. 

20.  Sensor  aperture  range-rate  measurement  offset  bias. 

21 .  Sensor  aperture  bearing  angle  measurement  offset  bias. 

22.  Sensor  aperture  bearing  angle  measurement  offset  bias. 

23.  Sensor  aperture  bearing  angle  measurement  scale  factor  bias. 

24.  Sensor  aperture  elevation  angle  measurement  offset  bias. 

25.  Sensor  aperture  elevation  angle  measurement  scale  factor  bias. 

26.  Sensor  aperture  roll  attitude/aperture  alignment  bias. 

27.  Sensor  aperture  pitch  attitude/aperture  alignment  offset  bias. 

28.  Sensor  aperture  yaw  attitude/aperture  alignment  offset  bias. 

29.  Sensor  field  of  regard. 

30.  Search  revisit  rate. 

31.  Track  maintenance  revisit  rate. 

32.  Number  of  correct  correlations. 

33.  Number  of  correct  non-correlations. 

34.  Number  of  incorrect  (false)  correlations. 
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35.  Number  of  incorrect  non-correlations. 

36.  In  a  gated  non-unique  assignment,  the  number  of  declared  tracks  that  are 
assignable  to  real  objects  meeting  reporting  criteria;  divided  by  the 
number  of  valid  declared  tracks. 

37.  In  a  gated  non-unique  assignment,  the  number  of  declared  tracks  that  are 
unassignable  to  real  objects  meeting  reporting  criteria;  divided  by  the 
number  of  valid  declared  tracks. 

38.  As  assessed  for  a  particular  object  meeting  reporting  criteria,  the  mean 
normalized  Chi-squared  statistic  of  the  track  assigned  to  that  object. 

39.  Reporting  responsibility  declaration  for  each  track  at  each  evaluation  time. 

40.  Reporting  responsibility  transition  time. 

41 .  Frequency  of  reporting  responsibility  transition. 


Page  29 
Version  1 .0 
October  25,  2001 


